Skip to content

feat(outbound): close the provider seam for every sender (B7) - #1000

Merged
jiashuoz merged 14 commits into
mainfrom
feat/sending-provider-closure
Sep 5, 2026
Merged

feat(outbound): close the provider seam for every sender (B7)#1000
jiashuoz merged 14 commits into
mainfrom
feat/sending-provider-closure

Conversation

@jiashuoz

@jiashuoz jiashuoz commented Sep 5, 2026

Copy link
Copy Markdown
Member

Slice B7 of the sending abuse prevention plan (ops PR #320): close the provider seam B5 opened. Stacked on #999 (B6); base is feat/sending-worker-cutover and will be retargeted to main once B6 merges.

What changes

The relay can no longer send without a token. outbound.SMTPRelay's exported Send* methods and outbound.Sender's Send/SendOnce/SubmitOnce* are deleted. The only socket-opening entry point is ProviderSubmitter.SubmitOnce(ctx, auth, envelope). A new tracked-closure test (internal/outbound/provider_authorization_guard_test.go) parses every production Go file and fails on any net/smtp import or any call to the relay's socket core outside the named exceptions, so a future bypass cannot land silently.

Notification paths cross the gate. hitlnotify (approval emails) and webhooknotify (webhook health notices):

  • the enqueue prepares a customer_notification operation in the same transaction as the source row (PrepareNotificationTx) and stamps it on the job (operation_ref);
  • the workers run the same Reserve → early hold → ConsumeAttempt → authorized submit order as the message worker, snoozing on a hold with no provider I/O;
  • a job from a pre-floor slot resolves its operation at fire time and stamps it once (jobs.StampJobArg, key-absent guard, existing fields preserved).

Public feedback mail (POST /api/feedback) is a public_feedback_notification operation keyed by a server-minted submission id; its envelope is the configured notify set, never the request. The in-request retry loop is bounded (4 attempts), every physical attempt is its own charged ordinal with its own attempt id on the wire, and a definite rejection or a lost acceptance stops the loop.

Operator command e2a -reconcile-legacy-sending-jobs: stamps an operation onto every pending outbound_send / hitl_notify / webhook_notify job that has none (through the exact Prepare path its enqueue would have used), cancels orphans whose source row is gone, leaves paused-account jobs to the worker's hold path, prints counts, and exits nonzero unless every scanned job was decided. Each job is one transaction; a failure leaves it untouched for a rerun.

Wiring: cmd/e2a/main.go, testutil.TestServer, and the contract server build the notifiers over the shared ProviderSubmitter and hand the API the submitter + gate (SetProviderSubmitter).

Review round 1 (correctness + adversarial, Opus) → fixes in the second commit

  • Notification operations are keyed by their source (op_hitl_<msg>, op_wh_<kind>_<webhook>_<episode>): PrepareNotificationTx is idempotent per hold / per health episode, and both notify workers cancel a job whose reference names any other operation — the binding the B6 message worker has.
  • Compose before any charge: Deliverer is now Compose + Submit; the notify workers run compose → Reserve → hold → ConsumeAttempt → Submit, so an owner-lookup / signing / DKIM failure costs no ordinal and the token is consumed immediately before the socket. The startup "notifier not wired yet" retry now also costs nothing.
  • Reconcile command re-reads each job under FOR UPDATE inside its own transaction and skips one a worker claimed or stamped meanwhile; output separates paused / skipped from remaining.
  • Feedback loop submits the token's canonical recipient set (an overlapping TO/CC config no longer fails every attempt), paces retries to fit the handler budget, keeps the real SMTP error on a deadline, releases a reserved attempt when authorize errors, and returns instead of panicking on an id-mint failure.
  • Closure guard: never skips (walks up to go.mod, git optional), matches method values / method expressions, exception is the SubmitOnce symbol not the file, asserts its sentinel is live, and fences the SES v2 SDK import to sender-identity provisioning.
  • Webhook health notices older than seven days are dropped instead of snoozing forever behind a paused account.
  • Wiring test for the three new composition-root edges (both notify bundles' gate, the API's submitter), plus StampJobArg tests for the internal/jobs coverage floor.

Review round 2 (mutation-tested re-review, Opus) → third commit

Every round-1 fix held under mutation. One blocker found in the fix commit itself: migration 113 (v1.8.7) stamped adopted notify jobs with op_<md5> references, which the new binding would have cancelled on any upgrade crossing v1.8.7. Now a reference that is not a derived id is treated as pre-derivation: the worker re-resolves it through the Prepare path and replaces it once (jobs.SetJobArg); the reconcile command scans and re-keys those too. Also: bounded the feedback attempt release (2s), symmetric 7-day age guard on HITL notices, episode key in microseconds, nil-receiver guards on the new compose path, doc/comment corrections, and the guard's residual scope stated in its comment.

Design-level, not changed here (needs a decision): public feedback charges the two global customer pools by spec (design §"public feedback consumes both global pools", A4 gate). The adversarial reviewer measured 8 probation units per request (4 attempts × 2 recipients) behind the 10/hr/IP limit, i.e. one IP can drain the 150/day probation pool in under 8 hours once budget_mode: enforce is armed. Options: a dedicated small feedback pool, a tighter per-IP limit, or a daily platform cap on feedback mail. Dormant while the policy is disabled.

Behaviour note: notification and feedback mail now carries X-SES-CONFIGURATION-SET, so SES publishes delivery feedback for it; it correlates to no message row and the SNS consumer acks it as unknown (a log line, no suppression).

Not in this PR

  • No public API change (no OpenAPI / SDK / CLI / MCP delta).
  • No policy activation; every environment still runs the disabled policy (pass-through admission, attempts durable).

Tests

  • internal/outbound: closure guard; relay tests over the unexported core.
  • internal/hitlnotify, internal/webhooknotify: gated-order worker tests (authorize-then-deliver, early/late hold and gate error snooze without I/O, legacy resolve+stamp once), notifier tests through the submitter, e2e harness over the gate.
  • internal/agent: feedback seam tests with a scripted SMTP server (retry is a new ordinal, 5xx not retried, lost acceptance not retried, bounded retries, envelope is configuration; wire carries the attempt header).
  • cmd/e2a: reconcile command (stamp live, cancel orphan, skip finalized and non-submitting kinds, stamped ref round-trips, second pass empty; undecided job reported and untouched, nonzero exit).

Local: fmt/vet clean; affected suites and -race on the three notify/outbound packages green. internal/e2e has known environmental failures on clean main locally; CI is the authority.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX

jiashuoz and others added 13 commits September 4, 2026 22:01
…asons

Two additive local failure reasons for the sending-protection holds:
submission.policy_budget_expired (a sending-budget hold reached its
seven-day deadline) and submission.sending_setup_expired (SES tenant
readiness did not land within the 72-hour setup deadline). Both are
local, correctable outcomes like submission.local_retries_exhausted, and
neither may ever be reported as a recipient rejection or a provider
outage.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
… paths

Two callers hold provider evidence but no token: the worker that finds
provider-accept evidence already recorded on a row it is about to
re-drive, and the terminal reconciler settling a stranded row from that
same evidence. Neither can name an ordinal. SettleOperation applies the
outcome to the latest attempt whose provider call started — never a
later ordinal that was only reserved, and nothing when no attempt ever
dialed — through the same body SettleProvider uses. LookupOperation
recovers a reference for an operation that already exists; it is not a
constructor, and every Gate method still reloads the row under lock.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The send claim now returns the message's persisted hold class and
anchor (migration 116) plus the owning account's last_resumed_at and
ses_tenant_ready_at, so every worker execution can re-derive the same
deadline. RecordOutboundHold writes the pair only while the message is
pre-terminal; every terminal write — sent, failed, evidence-settled,
trash-cancelled — clears it, so a stale hold can never outlive its
message's outcome. The two new local expiry reasons are recognized as
complete terminal fallbacks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The outbound send worker now authorizes every provider call through the
sending-protection Gate, in the fixed order the design names: Reserve
the durable attempt; snooze on an early hold without provider I/O;
DeferAttempt on a rate deferral and CancelAttempt on a final suppression
match; ConsumeAttempt as the last serialized decision; then the
authorized submitter, which redeems the token immediately before the
socket opens and settles the provider's answer. A later execution after
a confirmed attempt returns to Reserve, which allocates the next ordinal.

The worker-owned RampGate and agent.NewOutboundRampGate are removed: the
ramp is composed inside the gate and its progress moves only through
settlement. The Deliverer contract carries the token; the production
deliverer is outbound.ProviderSubmitter and refuses to dial without one.
A lost 250 (ErrProviderAcceptanceUnknown) is retried as a new ordinal and
never settled.

Enqueue prepares the operation in the accept transaction, between the
message insert and the River insert; a paused account is refused there
(ErrSendingPaused, HTTP 403 sending_paused) rather than queued. Jobs from
a pre-floor slot carry no reference and resolve at fire time through the
same Prepare path.

Finite holds persist a class and anchor on the message and derive the
deadline every execution: 72 hours for rate/ramp/provider and tenant
setup, seven days for policy budget. The first finite hold anchors at
the latest of accept, schedule, review, and last resume; a budget hold
promotes any class and keeps the anchor; policy_budget never changes
again; tenant readiness landing inside the setup deadline moves the
class to rate/ramp/provider exactly once; a pause has no clock but a
running deadline keeps running. Expiry emits the class's own reason.

Terminal reconciliation is settlement-only: an evidence-settled row also
settles the attempt that dialed through Gate.SettleOperation.

cmd/e2a gains one composition root (newOutboundSending) and a wiring
test that proves the registered send path holds the concrete gate and
the ProviderSubmitter-backed deliverer. The test servers build the same
composition with the disabled policy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
The machine-checked contracts caught three vocabularies the worker
cutover widened without saying so: the error-code catalog and the
ErrorBody.Code documentation (sending_paused, 403, auth family), the
lifecycle reason table in docs/api.md (submission.policy_budget_expired,
submission.sending_setup_expired), and the OpenAPI description the two
generated SDK models embed. Both SDK error maps classify sending_paused
as a non-retryable permission error, with tests.

The email-eval integration runner's job-args parser insisted on exactly
one key; the accept transaction now stamps operation_ref beside
message_id, so the parser admits that key and still rejects any other.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Two parallel reviews (correctness + adversarial) over the first cut.
Every item has a named test.

Contract surfaces (the blocker in both): the two new lifecycle reasons
join the hand-maintained reason_code enum tag, both closed-vocabulary
tests, the regenerated spec, both generated SDK models (mirrored by
hand; the generator needs Docker, and the description now carries no
apostrophe so the two generators agree), the web lifecycle parser and
timeline, docs/api.md and docs/events.md. sending_paused is registered
in the error catalog, the ErrorBody.Code doc, docs/api.md, and both SDK
error maps.

Worker:
- A gate outage is a bounded rate/ramp/provider hold, not an unbounded
  snooze.
- A paused job evaluates no deadline; a persisted deadline is not
  extended and the first hold after resume applies it.
- A provider outage never emits the setup reason (expiryReasonFor).
- markFailed's evidence-settle branch settles the dialed attempt, as the
  reconciler already did; a failed post-acceptance settlement is retried
  before it is logged as critical; a provider-id conflict is surfaced as
  an invariant alarm.
- The job's operation reference must name its own message; a mismatch
  cancels before any ledger call. Enqueue refuses a zero reference.
- HoldClassFor maps reasons by name; the armed worker RegisterJobs
  builds is exposed (Jobs.SendWorker) so the wiring test can prove it
  carries the gate and the legacy resolver and the submitter carries the
  configuration set.

Gate: SettleOperation prefers the oldest dialed attempt with no provider
id yet, so evidence arriving in send order binds each attempt when
several dialed.

Paused accounts on every enqueue path: 403 sending_paused on the direct,
platform-test, and HITL-approve paths; the TTL auto-approve sweep defers
a paused account's expired review by an hour (DeferReviewExpiry) instead
of re-picking it first every cycle and starving the batch.

The email-eval integration runner admits the operation_ref args key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Mutation-tested re-review of the previous fix round: no blockers, six
should-fixes.

- SettleOperation without a token resolves the attempt as: one already
  bound to this exact provider id (a replay stays home), else the oldest
  dialed attempt with no id, else the latest dialed. The earlier
  oldest-unbound-first rule let a replay for attempt one bind attempt
  two; the test now covers that shape.
- snoozeOnGateError threads the live reservation into the bounded hold,
  so an expiry at final authorization gives the attempt back instead of
  stranding it under an enforcing policy.
- MarkFailed returns the evidence's provider id, and the worker's
  evidence settle under a terminal write carries it — the reconciler
  already did. The two evidence paths now agree.
- resettle logs at critical level when the context ends mid-retry; a
  dedicated test covers the retry itself.
- The wiring test registers workers exactly as main does and inspects
  the worker River received (Jobs.RegisteredSendWorker), so a
  RegisterJobs that bypassed the armed constructor fails it.
- sending_paused is marked experimental beside blocked_by_policy in the
  stability extension, the docs, and the description, since the pause
  control ships disabled and pre-GA. The Python forward-compat table
  gains the two lifecycle reasons.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Slice B7 of the sending abuse prevention plan. The relay no longer exports
a send method; ProviderSubmitter.SubmitOnce with a gate token is the only
way to reach the provider, and a tracked-closure test parses every
production file to keep it that way.

- hitlnotify + webhooknotify: enqueue prepares a customer_notification
  operation in the source transaction and stamps it on the job; workers
  run Reserve -> early hold -> ConsumeAttempt -> authorized submit and
  snooze on a hold without provider I/O; pre-floor jobs resolve at fire
  time and are stamped once (jobs.StampJobArg).
- public feedback: server-keyed public_feedback_notification operation
  with a bounded per-attempt Reserve/Consume/Submit loop; a definite
  rejection or a lost acceptance stops it.
- e2a -reconcile-legacy-sending-jobs: stamps pending outbound_send /
  hitl_notify / webhook_notify jobs that carry no operation, cancels
  orphans, exits nonzero unless every scanned job was decided.
- main, TestServer and the contract server build the notifiers over the
  shared submitter and hand the API the submitter + gate.
- design addendum in docs/design/async-message-pipeline.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Review round 1 of the provider-seam closure (B7).

- PrepareNotificationTx derives the operation id from its source
  (op_hitl_<message>, op_wh_<kind>_<webhook>_<episode>), so a repeat
  preparation yields one operation and the notify workers cancel a job
  whose reference names any other operation.
- The notify Deliverer is Compose + Submit; workers run compose ->
  Reserve -> hold -> ConsumeAttempt -> Submit, so a compose failure
  charges nothing and the token is consumed right before the socket.
- Reconcile command re-reads each job under FOR UPDATE and skips one a
  worker claimed or stamped meanwhile; counts separate paused/skipped.
- Feedback loop submits the token's canonical recipients, paces retries
  to fit the handler budget, keeps the SMTP error on a deadline, releases
  a reserved attempt when authorize errors, no panic on id mint.
- Closure guard never skips, matches method references, exempts the
  SubmitOnce symbol only, asserts its sentinel, fences the SES v2 import.
- Webhook health notices older than seven days are dropped.
- Wiring test for the notification bundles and the API seam; StampJobArg
  tests for the jobs coverage floor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Re-review of the provider-seam closure (B7) found that migration 113
stamped adopted notify jobs with op_<md5> references, which the new
source binding would have cancelled on any upgrade crossing v1.8.7.

- A reference that is not a derived id is treated as pre-derivation: the
  notify workers re-resolve it through the Prepare path and replace it
  once (jobs.SetJobArg); a derived id for another source still cancels.
- The reconcile command scans and re-keys those references too.
- Bounded the feedback attempt release (2s); symmetric 7-day age guard on
  HITL notices; episode key in microseconds; nil-receiver guards on the
  compose path; doc and comment corrections; the closure guard states
  its residual scope.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
Round-3 re-review nits: COALESCE the conforming-reference predicate so a
reference with no id cannot fall out of a NOT scan, decode only the
source fields so such a reference is replaced rather than failing to
decode, and state in the workers that any non-derived shape re-derives
from the job's own source.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AjfGxvXW6fNKWGFHuo68yX
@jiashuoz
jiashuoz changed the base branch from feat/sending-worker-cutover to main September 5, 2026 18:36
…-closure

# Conflicts:
#	cmd/e2a/outbound_wiring.go
#	cmd/e2a/sending_policy_wiring_test.go
#	docs/design/async-message-pipeline.md
#	internal/testutil/contract_server.go
#	internal/testutil/server.go
@jiashuoz
jiashuoz merged commit 0420cc3 into main Sep 5, 2026
29 checks passed
@jiashuoz
jiashuoz deleted the feat/sending-provider-closure branch September 5, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant